Dynamic replacement of cinematic stage props in program content

ABSTRACT

Stage props, such as consumer products, in legacy program content are replaced with dynamically replaceable assets. Such assets can be used for advertising purposes in that images of different products and services can be inserted therein and are changeable via a network. Such assets may be created by identifying props in the content for replacement, and determining location and visual effects information for each identified prop. The content may be played back by broadcasting the content and reference information designating parameters for inserting the assets into the content. Replacement images, such as images of products, may be obtained via a network. The visual appearance of each replacement image may be modified using visual effects information so the inserted objects appear natural as if they were part of the original scene.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/371,367, filed Mar. 7, 2006, entitled “DYNAMIC REPLACEMENT OFCINEMATIC STAGE PROPS IN PROGRAM CONTENT,” the entire disclosure ofwhich is hereby fully incorporated by reference herein in its entirety.

This application is related to U.S. patent application Ser. No.11/371,215, filed Mar. 7, 2006, entitled “DYNAMIC INSERTION OF CINEMATICSTAGE PROPS IN PROGRAM CONTENT,” also by inventors Gary Zalewski andRiley R. Russell, the entire disclosure of which is incorporated hereinby reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to digital video processing, andmore specifically to digital insertion of images in program content foruse in advertising.

2. Discussion of the Related Art

One traditional form of advertising is the television commercial. Suchtelevision commercials typically consist of brief advertising spots thatrange in length from a few seconds to several minutes. The commercialsappear between shows and interrupt the shows at regular intervals. Thegoal of advertisers is to keep the viewer's attention focused on thecommercial, but often times the viewer will change the channel duringthe commercial to avoid watching the commercial.

Another way that viewers have avoided television commercials is by usingdigital video recorders (DVRs). With a DVR a person can record a showand then fast forward through the commercials during playback. Othermedia players are expected to provide viewers with even greater abilityto track and skip commercials.

It is with respect to these and other background information factorsthat the present invention has evolved.

SUMMARY OF THE INVENTION

The present invention advantageously addresses the needs above as wellas other needs by providing a method for use in modifying programcontent. The method comprises the steps of: identifying props in thecontent for replacement; determining location information for eachidentified prop; determining visual effects information associated witheach identified prop; and creating a reference file that includes thecontent and the location and the visual effects information associatedwith each identified prop.

Another embodiment of the present invention provides a system formodifying program content. The system comprises: means for identifyingprops in the content for replacement; means for determining locationinformation for each identified prop; means for determining visualeffects information associated with each identified prop; and means forcreating a reference file that includes the content and the location andthe visual effects information associated with each identified prop.

Another embodiment of the present invention provides a method for use ingenerating program content. The method comprises the steps of: providingcontent and one or more reference files that index location and visualeffects information associated with one or more replaceable props in thecontent; obtaining one or more replacement images for the props to bereplaced; modifying a visual appearance of each replacement image usingthe visual effects information; and inserting the one or morereplacement images in the content to replace the one or more replaceableprops.

Another embodiment of the present invention provides a system forplaying program content. The system comprises: means for providing thecontent and reference file that includes the location and visual effectsinformation associated with one or more replaceable props in thecontent; means for obtaining one or more replacement images; means formodifying a visual appearance of each replacement image using the visualeffects information; and means for inserting the one or more replacementimages in the content to replace the one or more replaceable props.

Another embodiment of the present invention provides a computer programproduct comprising a medium for embodying a computer program for inputto a computer and a computer program embodied in the medium for causingthe computer to perform steps of: accessing a reference file thatincludes program content and location and visual effects informationassociated with one or more replaceable props in the content; obtainingone or more replacement images; modifying a visual appearance of eachreplacement image using the visual effects information; and insertingthe one or more replacement images in the content to replace the one ormore replaceable props.

A better understanding of the features and advantages of the presentinvention will be obtained by reference to the following detaileddescription of the invention and accompanying drawings which set forthan illustrative embodiment in which the principles of the invention areutilized.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will be more apparent from the following more particulardescription thereof, presented in conjunction with the followingdrawings wherein:

FIGS. 1A, 1B, 1C, and 1D are pictorial diagrams illustrating examples ofdynamically replaceable stage props/assets in accordance withembodiments of the present invention;

FIGS. 2A and 2B are flow diagrams illustrating methods for creatingcontent with dynamically replaceable stage props/assets in accordancewith embodiments of the present invention;

FIGS. 3A, 3B, 3C, 3D, 3E and 3F are example pictorial diagramsillustrating dynamically replaceable stage props/assets in accordancewith embodiments of the present invention;

FIG. 4 is a flow diagram illustrating a method for creating content withdynamically replaceable stage props/assets in accordance with anembodiment of the present invention;

FIG. 5 is a flow diagram illustrating one method for playing backcontent with dynamically replaceable stage props/assets in accordancewith an embodiment of the present invention;

FIG. 6 is a system diagram illustrating an example implementation of aportion of the method shown in FIG. 5 in accordance with an embodimentof the present invention; and

FIG. 7 is a block diagram illustrating a computer system that may beused to run, implement and/or execute the methods shown and describedherein in accordance with embodiments of the present invention.

Corresponding reference characters indicate corresponding componentsthroughout the several views of the drawings.

DETAILED DESCRIPTION

When viewers avoid traditional television commercials as describedabove, the advertisers' messages are not received by the viewers, whichis disadvantageous. Because of such disadvantages, embodiments of thepresent invention involve the use of advertising in the television showor other content itself.

Embodiments of the present invention involve thereplacement/substitution of product placement advertisements in thecontent itself. That is, the content includes one or more assets, whichare locations in the video where images of different products can beinserted. For example, a cereal box that appears in an episode of acertain television show can be replaced with a different brand cerealbox. Typically, the company that sells the inserted cereal will pay forthis asset since it is a form of advertising. In this way each time theepisode of the show is aired the asset can be sold to the same or adifferent advertiser.

As will be described below, such assets can be identified in legacyprograms. That is, many older television programs and movies includeproducts and other stage props that may be outdated, no longer exist, beprovided by companies that no longer exist, etc. Many such olderprograms get replayed over and over again through reruns. Embodiments ofthe present invention provide methods for identifying the fixed stageprops in these legacy programs and converting them into dynamicallyreplaceable props that become valuable assets. This way, each time theprogram is rerun an image of a current product or service can beinserted into the asset, which can be used to generate advertisingrevenue.

In other embodiments of the present invention new program content isdeveloped with such advertising assets in mind from the beginning. Thisallows for the content to be developed to provide appropriate strategicpositioning and sizing of the assets. For example, a motion picturecompany can work closely with advertising experts in developing theassets during production of the motion picture.

In still other embodiments of the present invention new digitalprops/assets are inserted into legacy programs where no fixed proppreviously existed. That is, new props are added, such as for examplepictures on blank walls where the pictures contain some ad-affinity,brand, etc. Such embodiments help to solve the issue where legacycontent may have limited opportunity for product/stage prop replacement.Thus, for example, there is no need to replace a cereal box when one canbe added into the scene.

Once the content has been modified or developed to include thedynamically replaceable advertising assets, the images of variousdifferent products and services can be changeable over a network.

Identifying Assets in Legacy Content:

Legacy content includes content that was produced in the past, perhapsyears earlier. For legacy content, the first part of the process is togo through it and identify the assets that can be replaced. For example,such assets can include, but are not limited to, cereal boxes, soda cansand bottles, pizza boxes, other product boxes, cartons, containers,services, etc. The assets can include anything where a differenttexture, bitmap image, video stream, jpeg image, etc., of a differentbrand can be overlaid onto the existing object.

Referring to FIGS. 1A, 1B, 1C, and 1D, there is illustrated an exampleof how a fixed stage prop can be identified in legacy content and thenreplaced with a dynamically replaceable asset. Specifically, in FIG. 1Aa scene 100 in a legacy program may include a character 102 standingnext to a table 104. On the table 104 may be a fixed stage prop such asa box of cereal of a certain brand, such as for example the “1960s BESTCereal” 106 that is shown. Because the “1960s BEST Cereal” 106 may beoutdated and no longer available, it is desirable to replace it with acurrently available product. Doing so would increase the advertisingvalue of that stage prop making it a valuable asset.

In FIG. 1B the “1960s BEST Cereal” 106 has been replaced in the scene100 with a dynamically replaceable stage prop/asset 108. The dynamicallyreplaceable stage prop/asset 108 comprises data that is inserted intothe program content that allows different images of different productsor services to be inserted into the scene to replace the previous stageprop. The new images of the different products or services are what willbe seen by the viewer.

For example, in FIG. 1C a new and more modern brand of cereal called“NEW Century Cereal” 110 has been inserted into the asset 108 in thescene 100 to replace the “1960s BEST Cereal” 106. The program could berun for a period of time while the suppliers of “NEW Century Cereal” payfor this advertising opportunity. Then later, a different advertiser maywant to pay to have their product displayed in the asset 108. Forexample, in FIG. 1D a different product altogether, namely “Crispy GoodCrackers” 112, has been inserted into the asset 108 in the scene 100.The suppliers of “Crispy Good Crackers” may pay to have their productdisplayed for a certain period of time while the program is rerun, suchas for example just one episode or possibly a whole season or more.

Referring to FIG. 2A, there is illustrated a method 200 of creatingdynamically replaceable stage props/assets in accordance with anembodiment of the present invention. The method 200 begins in step 202in which stage props are identified in the content. The stage props thatare identified are those that can or those for which it would beadvantageous to be replaced with dynamically replaceable assets.

Specifically, in accordance with some embodiments of the presentinvention, the process of identifying such stage props/assets may beperformed manually by going through the content, i.e. on aframe-by-frame basis. With manual identification of the assets one goesthrough the content in a frame-by-frame or similar manner and identifiesthe assets. Nearly anything in the content may be used as an asset. Forexample, one might identify a pizza box or other product box orcontainer that appears in the content as a stage prop.

In step 204 the referential coordinates or other location information ofeach stage prop/asset in the content are determined and recorded overtime. That is, location information is determined for each identifiedstage prop that is to be replaced. For example, after an asset isidentified, the content may be rewound back to the first instance of theasset. The asset is then zoomed in on and the referential coordinates ofthe asset are identified manually.

For example, one of the corners of the asset, such as a box, may beclicked on. Any corner may be clicked on, such as the upper left orlower right corner. An overlay region may then be identified for propreplacement in the perspective for the particular frame. Because thecameras are usually moving, one will typically manually go through eachframe and plug in the coordinates that define the location of the boxwith respect to the camera.

The manual identification of the stage props/assets does not have to bedone on a frame-by-frame basis. Namely, it can be done every otherframe, every several frames, or nearly any number of frames. Forexample, the corners of the box can be identified in one frame and thenmany frames can be skipped until the position of the box changes. Theframes can be skipped because the reference coordinates of the box willbe the same as the previous frame.

Automated smoothing functions may be applied after manual identificationprocess to assure that the insertion regions are bound properly andconsistently, without jitter and with proper aspect ratios. Suchsmoothing functions may ensure that the present invention operatestransparently to achieve the targeted psycho-visual experience for theend user. Other functions including those that result in jitter,strobing, halo, etc. may be applied to bring contrast or attention tothe replacement prop.

In addition to tracking the referential coordinates identified for eachstage prop/asset across frames, the idea, theme, concept, etc.,associated with each asset may also be identified or designated. Sceneinformation, such as light sources, diffusers, reflectors, etc., theirtypes, locations, colors, intensities, etc. may be recorded in atime-based reference file. Other characteristics of each asset may alsobe identified or designated. This data is saved along with thecoordinates.

All or some of the data may be used to develop a function that over thetime the content is running provides appropriate referential coordinatesof the location in the video of the asset so the replacement image canbe calibrated to smooth the insertion of the replacement content intothe media.

Determining after Effects in Legacy Content:

After the stage props/assets are identified, the after effects of theassets are then determined. Specifically, in step 206 the after effectsor other visual effects information of each stage prop/asset in thecontent are determined and recorded over time. That is, visual effectsinformation associated with each identified stage prop are determined.Such after effects or visual effects information may include renderingfunctions, variables and settings. After effects are applied accordingto scene information so the image psycho-visually complies with thescene environment in a proscribed manner. That is, the after effects ofthe original asset are determined so that the data can be used to makethe replacement image, in most cases, fit in, blend with the scene andlook natural.

Such after effects may include not only the coloring, shading, focus,sharpening, blurring aspect, etc., but also any other or all visualqualities that were applied to the original asset in order to match theoptical qualities. Such visual qualities may, for example, include theluminescent, reflectivity, and/or shading qualities. Rendering settingsare typically set to achieve visual qualities that reflect theenvironment of the real, original scene.

It is important to understand what after affects would have to beapplied to an image to blur it or make it fit in with the originalqualities of the asset being replaced. A profiling step may be used inwhich, for example, a soda bottle in the background is viewed todetermine the magnitude of the blur level or the blur threshold.

A post filter may be used to try to match a new texture to give theoptical quality of the original object. The optical quality will oftenchange on a frame-by-frame basis. One way to check this is to check theoptical quality of the asset in the first frame, and then check theoptical quality of the asset in a later frame to confirm that it has thesame blur level or same optical filter level. Different frames can bechecked until the asset is no longer in the view of the camera.

One goal is to develop a function that over the time the content isrunning dictates what after effects or other visual effects informationhave to be applied to a new texture that is dynamically served into thecontent to replace a previously fixed stage prop when the content isplayed. Such after affects or other visual effects information may bedetermined over time that the content is running.

Automated Identification of Assets and after Effects in Legacy Content:

In accordance with other embodiments of the present invention, theprocesses of identifying assets and/or after effects of assets in legacycontent may be at least partially automated. For example, a softwaretool that is designed for these tasks may be used. Namely, part of theprocess of processing a legacy video, television show, movie, or othercontent, could be a tool phase where a software tool is used to processthe content.

Referring to FIG. 2B, there is illustrated a method 220 that may beperformed by such a tool in accordance with an embodiment of the presentinvention. In step 222 the user clicks on the stage prop in the content.That is, such a tool allows the user to click on the asset in the legacycontent, such as a cereal box. The tool zooms in on the box andidentifies the referential coordinates or other location information forthe box. In step 224 the tool automatically tracks and saves thelocation information of the stage prop as the program content plays.

The tool may also be configured to determine the visual effectsinformation associated with the box. In step 226, the tool automaticallytracks and saves this information for the stage prop as it moves duringthe playing of the content.

Furthermore, in some embodiments of the present invention, the softwaretool also profiles the box. That is, after the box is clicked on, thetool makes a profile of it, and then the user can decide on shading,blur, etc.

The tool can be configured to accept input information about the box aspart of the profiling process. Thus, in step 228 the tool accepts inputinformation about the stage prop from the user. That is, the user canprovide the tool with all the information the user has about the box inthe legacy content. For example, the user can provide the tool with thetype of box it is, i.e. the specific legacy product that was packaged inthe box. The tool will look up the image for that specific box and willlook at the pixel settings that represent the shading in that filmenvironment. The tool will then be able to determine the generallighting principles associated with the particular bitmap image that itknows is in that location and orientation. The tool then uses all ofthis information to automatically do an analysis of the stage prop imageand generate a profile, which is indicated by step 230.

The generated profile can indicate where the rendered asset should beshaded and where the light source should be positioned based on thecamera view of the box. The profile may also include information forrendering the inserted object or predict a light source over the courseof the changing camera environment. It can also show that the grading ofthe light is generally falling on the box in a particular pattern. Theuser can review all of the information in the profile and makeadjustments.

The generated profile helps the user go through a calibration processfor a legacy show to help find and determine all opportunities fordynamic replacement of legacy assets with advertising content.

Identifying Assets in New Content:

As was mentioned above, in other embodiments of the present inventionnew program content is developed with such dynamically replaceablecinematic stage props and advertising assets in mind from the beginning.This allows for the content to be developed to provide appropriatestrategic positioning and sizing of the assets. For example, a motionpicture company can work closely with advertising experts in developingthe assets during production of the motion picture.

With new program content, the location of stage props, scene informationand visual effects information such as after affects of the assets willalready be known or can be captured during filming.

During film production a set of props can be used to automate thetracking of location of the prop and define the profile for the asset,based on scene information and the manipulation of the prop in thescene.

Passive or active props or a combination thereof may be used in thepresent invention.

Props are preferably but not necessarily shaped with a form factor toresemble the category of prop they represent (i.e. cereal box, soda can,cracker box, bottle, picture on wall, etc.).

Passive props may distinguish themselves from the scene environment toenable the present invention to automatically track the prop through ascene and in a manner that allows the system to record scene informationthat is reflected on the prop. Passive props may be made of green screenmaterial, retro-reflective material combinations or suitable. Props maybe encoded with visual markers, indexed visual markers or similar so theprops can be referenced and tracked in the scene. For example, the X, Y,Z coordinates for the center of each of those props or the perimeterprofile of the props can be tracked in an automated sense duringanalysis of the video frames with the present invention and matched witha definition of each of the props. Further processing may fit the proptracking records with the profile of the prop to conform aspect ratios,smooth, etc.

Visual effects and after effects of the assets can be defined based onthe known scene information, including time-based camera and proppositioning information. Given the scene information and determinationof the position of the stage prop and the position of the camera,lights, etc. at any instant in time, the present invention has theappropriate inputs to compute appropriate after effect settings for theinserted stage prop.

Passive stage props may reduce or eliminate the need for the manualcalibration process of going through the film frame-by-frame to identifythe location of the asset tag. For example, a passive stage prop sodabottle i.e. made with retro-reflective markers is presented in thescene. As the soda bottle moves around in a character's hand, thetime-based position information for the soda bottle is automaticallydetermined by machine vision tracking methods known in the art andrecorded. The position information may include orientation, tilt, yawland roll metrics or suitable information to yield appropriatereplacement regions for the soda prop/asset. Later, new props and/orreplacement textures may be rendered into the frames where the sodabottle exists to effectively change the soda bottle from Coke® to Pepsi®or to a new brand that enters the market after the scene was produced.

In addition to passive props, active props may contain electroniccircuits and systems that allow the prop to intelligently sense theenvironment in which they operate. Such sensors may enable global orlocal positioning capability and other sensors that record theenvironment in which the prop is exposed during a scene. Such sensorsmay include but not be limited to, photodector(s), photodetectorarray(s), CCD sensor(s), CCD sensor array, accelerometer, etc.

Active props may record or transmit time based telemetry data thattracks the prop in the scene along with scene information and sceneinformation directed at the prop including reflections, shadings,lightings, etc.

Active props may be tracked in a variety of ways, including theirposition, orientation, tilt, yawl, roll, etc. Props may also be trackedfor deformations of form. Props may be tracked to understand how theyare manipulated and deformed, so replacement objects comply with thesame deformation. For example, a cereal box or toothpaste prop may betracked to determined how its shape is deformed in the scene by theactor who may manipulate the prop by squeezed, opening, closing,accessing the prop, etc.

Active props can record or transmit time-based reflections of objectsthat reflect onto the prop. Reflection data and other sensor data isused to configure the after effect settings for the prop at thetime-index of the video frame. When mixing the new textures on the prop,the new texture is rendered with the after effect settings so the newtexture can appear on the prop object with actual reflections from thescene.

Thus, with new program content the advertising assets can be developedalong with the content and the manual process described above may not beneeded.

Creation of Reference File:

After the assets are identified and the after effects are determined, animage or reference file is created. Specifically, in example step 208(FIG. 2A) a reference file is created that includes the original programcontent along with the location and after effects data relating toidentified replaceable stage props/assets. The reference file ispreferably time-based so that a video frame can reference the locationof the stage props and the after effect settings according to the frame.

The reference file does not need to bundle the original program content.The original content may already be available to an end user for exampleon a DVD, HD, memory card. In these cases, a user may download or streama reference file that contains the time-based stage-prop location andafter effect data but does not contain original content. The presentinvention may also operate in a mode where the user reference file isprovided, and the file itself contains the time-based stage-proplocation and after effect data along with the replacement ad texturesbut no original content. The present invention may also operate in amode where the user reference file is provided, and the file itselfcontains the time-based stage-prop location and after effect data alongwith the replacement ad textures and the original content.

As mentioned above, the image or reference file preferably includestime-based information that allows a player to recognize whenreplacement stage props are present in a scene, their profile, locationand rendering settings.

Valuing the Impression for Each Asset:

In accordance with another embodiment of the present invention, there isprovided a method of valuing the impression for each advertising asset.Specifically, in step 210 a value is determined and assigned to eachdynamically replaceable stage prop/asset in the content. That is, foreach asset, such as for example a green screen cereal box, there isprovided a mechanism for qualifying or valuing the quality of theimpression and for tracking qualities of the impression. Determining thevalue of each asset will help to influence the price to be paid by apotential advertiser.

The value of each asset can depend on many different factors. Suchfactors may include, but are not limited to, the size of the proprelative to the size of the screen, the amount of time that the prop isvisible, the amount of clarity or blur effect that is applied to theprop, etc.

Regarding the amount of clarity or blur effect that is applied to theprop, a soda bottle, for example, may be way in the back of program andit might be kind of blurry. Because of the focal length of the camerathe soda bottle may not be in focus, whereas the main character of theprogram is in focus. Thus, the amount of focus on the soda bottle is afactor. Because the soda bottle is somewhat blurry that might feed animpression formula that would value that advertising asset slightly lessthan if the soda bottle was right in front of the camera and in focus.

The amount of clarity or blur effect depends on whether the image needsto be blended or blurred in the after effect post process to blend it inwith the rest of the material. This is identified in the manual processdescribed above. In the case of the automated green screen prop process,the amount of clarity or blur effect may be automatically calculatedbased on the focal length of the camera, the type of lens, the distanceto the prop, etc.

Another factor that can be used in valuing an asset can be whether amain character touches the asset or draws attention to it in some way.The position and orientation of the asset can also be a factor. Forexample, a soda bottle might be positioned at an angle that mightdevalue the asset because it is not at its proper orientation just byway of the artistic design. Such design choices in the creation of thecontent can have an impact on the exposure and thus the value.

The value of the assets is a balancing of the reality and the artisticfreedom of a producer with the ability then to add value to those propsand change them. Based on all the different factors potentialadvertisers can ultimately competitively bid for advertising on thoseassets, as will be discussed below.

Adding New Assets to Legacy Content:

In accordance with other embodiments of the present invention, newdigital props/assets are inserted into legacy programs where no fixedprop previously existed. That is, dynamically replaceable stageprops/assets are established in video, web, program content in blank,open or convenient locations. These embodiments help to solve the issuewhere legacy content may have limited opportunity for product/stage propreplacement.

Referring to FIGS. 3A, 3B, 3C, 3D, 3E and 3F, there are illustratedexamples of how dynamically replaceable props/assets can be establishedin program content in accordance with such embodiments of the presentinvention. The program content may comprise a legacy program.Specifically, in FIG. 3A there is illustrated a scene 300 in a programthat may include a character 302 standing next to a table 304. Unlikethe scenario described above, there is no fixed stage prop on the table304.

In FIG. 3B a dynamically replaceable stage prop/asset 306 representing abeverage can has been inserted into the scene 300 and positioned so thatit is resting on the table 304. There was previously nothing shown onthe table 304, so the insertion of the asset 306 makes valuable use ofthe previously blank space in this program.

The dynamically replaceable stage prop/asset 306 can now be sold toadvertisers who want to insert their products into the program. Forexample, in FIG. 3C a brand of soda called “RED Soda” 308 has beeninserted into the asset 306 in the scene 300. The program could be runfor a period of time while the suppliers of “RED Soda” pay for thisadvertising opportunity. Then later, a different advertiser may want topay to have their product displayed in the asset 306. For example, inFIG. 3D a different product, namely “BLUE Pop” 310, has been insertedinto the asset 306 in the scene 300. The suppliers of “BLUE Pop” may payto have their product displayed for a certain period of time while theprogram is rerun, such as for example just one episode or possibly awhole season or more.

FIG. 3E illustrates that a dynamically replaceable stage prop/asset 314representing a picture or poster can has been inserted into the scene300 and positioned above the table 304. There was previously nothingshown on the wall, so the insertion of the asset 314 makes valuable useof the previously blank space. This dynamically replaceable stageprop/asset 314 can also be sold to advertisers who want to insert theirmessage into the program. For example, in FIG. 3F a message regarding abrand of apparel called “Action Sports Apparel” 316 has been insertedinto the asset 314 in the scene 300. In this way new props/assets can beadded, such as for example pictures on blank walls where the picturescontain some ad-affinity, brand, etc.

Therefore, in these embodiments of the present invention new dynamicallyreplaceable stage props/assets are added to the program in open orconvenient locations. For example, if there is no cereal box in theoriginal scene in the legacy program, one can be added into the scene.

Referring to FIG. 4, there is illustrated a method 400 of adding newdynamically replaceable stage props/assets to program content inaccordance with an embodiment of the present invention. In step 402open, blank, or convenient locations in the content where dynamicallyreplaceable assets can or are desired to be inserted are identified. Instep 404 dynamically replaceable assets are established in one or moreof the identified locations.

The above-described steps of saving location and visual effectsinformation for the added assets can also be applied here where a newasset is inserted. Thus, in step 406 location information, such as thereferential coordinates, for each new asset in the content is determinedand recorded for each instant of time. Similarly, the processes ofrecording visual effects information and blending after effects can alsobe used. Therefore, in step 408 visual effects information of each assetin the content is determined and recorded for each instant of time. Withrespect to blending after effects and visual effects information,because there was no original fixed prop in the program, such blendingwill be to make the best fit and look with the rest of the scene.

In step 410, similar to as described above, a reference file is createdthat includes the original program content along with the dynamicallyreplaceable stage props/assets. And in step 412 a value is determinedand assigned to each dynamically replaceable stage prop/asset in thecontent.

Playback, Insertion of Replacement Images, and Networking:

Referring to FIG. 5, there is illustrated one example method 500 ofplaying back program content having dynamically replaceable stageprops/assets in accordance with an embodiment of the present invention.

When the reference file is ultimately played back such as by downloadinga TV show or playing a movie, it must be decided what replacement imageswill be inserted into the dynamically replaceable stage props/assets.

Many schemes exist to allocate advertisement objects to the replacementprops. Ads may have weights and a system may allocate ads to appropriatethe genre or specific titles to accommodate requested number ofimpressions sought by an advertiser, etc.

Based on all the different factors potential advertisers can ultimatelycompetitively bid for advertising in the assets. Thus, in step 502 ofthe method 500 advertisers bid for the assets. Such bidding may be basedon the determined value of the assets.

There are many different ways such bidding can be done, such as forexample by an auction or the like where the bidders are presented theprogram content and the types of assets available. For example, if oneof the assets is a cereal box, then companies wanting to advertisecereal will bid for that particular show and asset.

Once it is decided what replacement images will be inserted into thedynamically replaceable stage props/assets, in this example, the contentcan then be broadcast using the reference file, as is indicated by theexample step 504. As described above, in the present example, the imageor reference file includes the program content as well as a functionthat for each instant of time provides the location and visual effectsinformation for each dynamically replaceable stage prop/asset. As theprogram is played the reference file provides the referentialcoordinates of the location of each asset and the after affects thathave to be applied to a new texture that is dynamically retrieved from astorage means using communications methods known in the art.

In step 506 of the example method the reference file obtains replacementimages, objects or textures for the assets via a network. Morespecifically, when the media is played back the reference file willcommunicate with a network server and accept a new image, object ortexture for each ID. Such IDs identify the dynamically replaceable stageprops/assets.

In step 508 the replacement textures are passed through the appropriateafter effects for each asset in the content. After the new textures arepassed through the appropriate after affects they are merged into thecontent. This is illustrated by step 510 in which the replacementtextures are digitally inserted into the assets in the content on anational, local, group, or individual basis. That is, the completedmedia is created by downloading the image of the replacement prop anddigitally inserting the prop into the content.

The network server may comprise an advertisement server. Images of thereplacement props are dynamically served from the ad server. Existing adservers used for advertising in video games in entertainment systems maybe modified for this purpose. Namely, such existing ad serversdynamically serve ads into billboards in video games. Thatinfrastructure may be configured to dynamically serve the replacementprops into the assets identified in the content.

Step 510 provides that the replacement textures are digitally insertedinto the content on a national, local, group, or individual basis. FIG.6 illustrates an example of how this can be done in accordance with anembodiment of the present invention. Namely, a network 600 provides theprogram content to a number of cable companies, such as the cablecompany 602. The replacement prop insertion may be done iteratively,nationally or globally by the network 600 such that the same or mixedbundle replacement props appear in all local broadcasts. Alternatively,for programs that are broadcast nationally or globally, differentreplacement props and commercial textures may be inserted into thecontent in different local areas. For example, the replacement propinsertion may be done by local cable companies or local syndicates. Thelocal cable company 602 may insert the same prop that is insertednationally, or a prop having local significance may be inserted.

In another embodiment, different replacement props and commercialtextures may be inserted into the content depending on a type or groupof viewers. For example, one or more textures may be inserted for onlythe end viewers 604. In other embodiments, different replacement propsand commercial textures may be inserted into the content for only aspecific, individual viewer. An example of this scenario is illustratedfor the end viewer 606 where the ad insertion is performed by theviewer's cable box, entertainment system, or the like. The programcontent can be downloaded to the viewer's cable box or entertainmentsystem and then the ad insertion performed.

Profiling can be used for the scenario where the replacement propdepends on the type of viewer. Such profiling can utilize a user profilefor each viewer. For example, for viewers that are hockey fans, theymight see hockey props in the program content. Thus, the props and thestyle of the props are dependent on the profile of the user.

Therefore, there are many different ways that the network can flow theads into the media.

The methods and techniques described herein may be utilized, implementedand/or run on many different types of computers, graphics workstations,video game systems, DVD players, DVRs, media players, home servers,video game consoles, cell phones, portable devices, wireless devices andthe like. Referring to FIG. 7 there is illustrated such a system 700that may be used to implement and/or run the methods and techniquesdescribed herein. The system 700 includes a central processing unit(CPU) 702, a graphics processing unit (GPU) 704, digital differentialanalysis (DDA) hardware 706, a random access memory (RAM) 708, a massstorage unit 710, such as a disk drive, and a display monitor 712.

The CPU 702 and/or GPU 704 can be used to execute the steps of themethods and techniques described herein, and the various program contentand images can be rendered on the display monitor 712. Removable storagemedia 714 may optionally be used with the mass storage unit 710, whichmay be used for storing code that implements the methods and techniquesdescribed herein. However, any of the storage devices, such as the RAM708 or mass storage unit 710, may be used for storing such code. Eitherall or a portion of the system 700 may be embodied in a device 716, suchas for example a computer or video game console or system. Simulationsand/or processing may be run concurrently with other graphics tasks. Ona multi processing core, simulations can be executed on a separateprocessor.

While the invention herein disclosed has been described by means ofspecific embodiments, examples and applications thereof, numerousmodifications and variations could be made thereto by those skilled inthe art without departing from the scope of the invention set forth inthe claims. For example, the replacement images may be received over anetwork or through some other means, including DVD, memory card, etc. Adtextures and replacement images may include a model, image orcombination(s). Program content and reference file may be streamed. Theoriginal program content may include the original program content and/ora modified encoded version of the original program content. The presentinvention may also be used to insert or replace stage props intraditional commercial advertisements themselves. The present inventioncan be used for a variety of purposes, including new video immersivegame and entertainment models, messages, controls, tickers, news, framesare immersively placed and optimized for time-based scene specificplacement, etc. The present invention may be used to insert or replaceprops in commercial banner ads placed in traditional distributionchannels, including broadcast, web, network tv, games, etc. It should bewell understood that ‘content’ may include, but is not limited to,television shows, programs, films, movies, videos, websites,commercials, documentaries, video games, etc.

The present invention may be configured to upload feedback records totrack media use and user exposure to the advertisements. Ad models,textures may be buffered in storage on a device having intermittednetwork access. Ads may be placed on rotation schedules. The referencefile may be one or more files and contain other elements and objects andlocation data and after effect data may be separated in physicallydifferent files or streamed over a network over different channels—thisis still referred to as “reference file”. The present invention may beused on portable devices, cell phones, etc.

Therefore, in some embodiments stage props, such as consumer products,in legacy program content are replaced with dynamically replaceableassets. Such assets can be used for advertising purposes in that imagesof different products and services can be inserted therein and arechangeable via a network. Such assets may be created by identifyingprops in the content for replacement, determining location and visualeffects information for each identified prop, and creating a referencefile that includes the content and the location and the visual effectsinformation associated with each identified prop. A value may bedetermined for each asset or advertisers may bid on the assets to yieldadditional advertising revenue channel for the content. The content maybe played back by broadcasting the content and reference informationdesignating parameters for inserting the assets into the content.Replacement images, such as images of products, may be obtained via anetwork. The visual appearance of each replacement image may be modifiedusing visual effects information so the inserted objects appear naturalas if they were part of the original scene.

What is claimed is:
 1. A non-transitory computer readable storage mediumstoring one or more computer programs adapted to cause a processor basedsystem to execute steps comprising: identifying stage props displayed incontent; looking up a reference image associated with each identifiedstage prop; performing an analysis of each reference image and itsassociated stage prop to determine a lighting condition of eachidentified stage prop; and creating reference information configured todefine dynamically replaceable assets in the content by providing dataneeded to replace each identified stage prop with a replacement image asthe content plays; wherein the data needed to replace each identifiedstage prop with a replacement image comprises information indicatinglighting to be applied to the replacement image based at least in parton the lighting condition of the identified stage prop.
 2. Thenon-transitory computer readable storage medium of claim 1, wherein theidentifying stage props displayed in content comprises: receiving firstinput information that identifies a first stage prop displayed incontent.
 3. The non-transitory computer readable storage medium of claim2, wherein the first input information is generated in response to auser clicking on the first stage prop displayed in content.
 4. Thenon-transitory computer readable storage medium of claim 1, wherein thelooking up a reference image associated with each identified stage propcomprises: receiving second input information about a first stage prop;and looking up a reference image associated with the first stage propbased on the second input information.
 5. The non-transitory computerreadable storage medium of claim 4, wherein the second input informationis provided by a user.
 6. The non-transitory computer readable storagemedium of claim 1, wherein the one or more computer programs are furtheradapted to cause a processor based system to execute steps comprising:tracking and saving location information for each identified stage propover time as the content plays.
 7. The non-transitory computer readablestorage medium of claim 1, wherein the one or more computer programs arefurther adapted to cause a processor based system to execute stepscomprising: tracking and saving visual effects information for eachidentified stage prop over time as the content plays.
 8. Thenon-transitory computer readable storage medium of claim 1, wherein thereference information comprises information that can be used to make thereplacement image for each identified stage prop blend in with a scenein the content and look natural.
 9. The non-transitory computer readablestorage medium of claim 8, wherein the reference information comprisesinformation that can be used to make the replacement image for eachidentified stage prop appear to include actual reflections from thescene when the replacement image is rendered using the information. 10.The non-transitory computer readable storage medium of claim 8, whereinthe reference information indicates where the replacement image for eachidentified stage prop should be shaded.
 11. The non-transitory computerreadable storage medium of claim 8, wherein the reference informationindicates where a light source should be positioned based on an analysisof at least one reference image and its associated stage prop.
 12. Thenon-transitory computer readable storage medium of claim 11, wherein thereference information further indicates where the light source should bepositioned over a course of a changing camera environment for eachidentified stage prop.
 13. The non-transitory computer readable storagemedium of claim 11, wherein the reference information further indicatesa pattern in which a grading of light is falling on each identifiedstage prop.
 14. A method, comprising: identifying stage props displayedin content; looking up a reference image associated with each identifiedstage prop; performing, by a processor based apparatus, an analysis ofeach reference image and its associated stage prop to determine alighting condition of each identified stage prop; and creating referenceinformation configured to define dynamically replaceable assets in thecontent by providing data needed to replace each identified stage propwith a replacement image as the content plays; wherein the data neededto replace each identified stage prop with a replacement image comprisesinformation indicating lighting to be applied to the replacement imagebased at least in part on the lighting condition of the identified stageprop.
 15. The method of claim 14, wherein the identifying stage propsdisplayed in content comprises: receiving first input information thatidentifies a first stage prop displayed in content.
 16. The method ofclaim 15, wherein the first input information is generated in responseto a user clicking on the first stage prop displayed in content.
 17. Themethod of claim 14, wherein the looking up a reference image associatedwith each identified stage prop comprises: receiving second inputinformation about a first stage prop; and looking up a reference imageassociated with the first stage prop based on the second inputinformation.
 18. The method of claim 17, wherein the second inputinformation is provided by a user.
 19. The method of claim 14, whereinthe reference information comprises information that can be used to makethe replacement image for each identified stage prop appear to includeactual reflections from a scene when the replacement image is renderedusing the information.
 20. The method of claim 14, wherein the referenceinformation indicates where the replacement image for each identifiedstage prop should be shaded.
 21. A system, comprising: a processor basedapparatus that is configured to execute steps comprising: identifyingstage props displayed in content; looking up a reference imageassociated with each identified stage prop; performing an analysis ofeach reference image and its associated stage prop to determine alighting condition of each identified stage prop; and creating referenceinformation configured to define dynamically replaceable assets in thecontent by providing data needed to replace each identified stage propwith a replacement image as the content plays; wherein the data neededto replace each identified stage prop with a replacement image comprisesinformation indicating lighting to be applied to the replacement imagebased at least in part on the lighting condition of the identified stageprop.
 22. The system of claim 21, wherein the identifying stage propsdisplayed in content comprises: receiving first input information thatidentifies a first stage prop displayed in content.
 23. The system ofclaim 22, wherein the first input information is generated in responseto a user clicking on the first stage prop displayed in content.
 24. Thesystem of claim 21, wherein the looking up a reference image associatedwith each identified stage prop comprises: receiving second inputinformation about a first stage prop; and looking up a reference imageassociated with the first stage prop based on the second inputinformation.
 25. The system of claim 24, wherein the second inputinformation is provided by a user.
 26. The system of claim 21, whereinthe reference information comprises information that can be used to makethe replacement image for each identified stage prop appear to includeactual reflections from a scene when the replacement image is renderedusing the information.
 27. The system of claim 21, wherein the referenceinformation indicates where the replacement image for each identifiedstage prop should be shaded.